1. Field of the Invention
This invention relates generally to continuous processing systems that process streaming data, and, more specifically, to measuring latency in a continuous processing system.
2. Description of the Background Art
A continuous processing system processes streaming data. It includes statements (such as queries), written by programmers, which operate continuously on input data streams and which publish to output data streams.
When statements written by programmers are compiled, an execution graph may be created, where the execution graph is comprised of connected primitives that correspond to the compiled statements. An execution graph in a continuous processing system specifies the path for processing messages in accordance with the statements. In other words, the continuous processing system processes messages by “pushing” them through the execution graph.
For performance measurements, it is desirable to be able to measure latency in a continuous processing system. In particular, it is most desirable to be able to measure the total time it takes for a message generated by an “input adaptor” in the continuous processing system to travel though the execution graph and reach an “output adaptor.” An input adaptor receives data from an external input source and conditions the data for processing by the continuous processing system. Data processed by an input adaptor is published as rows of messages into one or more data streams. Such data streams are input to a query processor. If input data to the query processor generates output (by nature of the particular query being run), the query processor publishes such output to one or more output data streams. An output adaptor subscribes to one or more output data streams and conditions data in such streams for an external output destination.
A known method for measuring latency is to add tracking information to standard data messages. However, the problem with adding tracking information to data messages is that, data messages, by their nature are filtered, transformed, delayed or discarded on their way through the execution graph. In general, it is impossible to know if a particular input data message will generate an output message (especially when the execution graph includes a filter). This unpredictability makes calculating latency and detecting processing stalls very difficult.
Consequently, there is a need for a method for calculating latency that bypasses the unpredictability of the execution graph and allows for latency to be calculated reliably and easily.